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Abstract. In this paper we consider the restriction of a unitary irreducible representation of 
type Aq (A) of GL{4, M) to reductive subgroups H which are the fixpoint sets of an involution. 
f-H ' We obtain a formula for the restriction to the symplectic group and to GL{2, C), and as an 

, application we construct in the last section some representations in the cuspidal spectrum 

of the symplectic and the complex general linear group. In addition to working directly 
with the cohmologically induced module to obtain the branching law, we also introduce the 



J2 ' useful concept of pseudo dual pairs of subgroups in a reductive Lie group. 
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^ ! 1 Introduction 

. Understanding a unitary representation vr of a Lie groups G often involves understanding its 

00 ! restriction to suitable subgroups H. This is in physics referred to as breaking the symmetry, 

' and often means exhibiting a nice basis of the representation space of vr. Similarly, decomposing 

a tensor product of two representations of G is also an important branching problem, namely 
^ I the restriction to the diagonal in G x G. Generally speaking, the more branching laws we know 

^ ' for a given representation, the more we know the structure of this representation. For example, 

when G is semisimple and K a maximal compact subgroup, knowing the ii'-spectrum, i.e. the 
collection of K-types and their multiplicities, of vr is an important invariant which serves to 
describe a good deal of its structure. It is also important to give good models of both vr and its 
explicit -fC-types. There has been much progress in recent years (and of course a large number of 
more classical works, see for example (271 El [3 [8] ) , both for abstract theory as in [TT l [T2 l [HI [T3] . 
and concrete examples of branching laws in |24[ [25l [TSl HI [28] . 

In this paper, we shall study in a special case a generalization of the method applied in [19] and 
again in [8] ; this is a method of Taylor expansion of sections of a vector bundle along directions 
normal to a submanifold. This works nicely when the original representation is a holomor- 
phic discrete series for G, and the subgroup H also admits holomorphic discrete series and is 
embedded in a suitable way in G. The branching law is a discrete sum decomposition, even 
with finite multiplicities, so-called admissibility of the restriction to H; and the summands are 
themselves holomorphic discrete series representations for H. Since holomorphic discrete series 
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representations are cohomologically induced representations in degree zero, it is natural to at- 
tempt a generalization to other unitary representations of similar type, namely cohomologically 
induced representations in higher degree. We shall focus on the line bundle case, i.e. the j4q(A) 
representations. In this case T. Kobayashi [13] obtained necessary and sufficient conditions that 
the restriction is discrete and that each representation appears with finite multiplicity, socalled 
admissibility of the representation relatively to the subgroup. Using explicit resolutions and 
filtrations associated with the imbedding of H in G, we analyze the derived functor modules 
and obtain an explicit decomposition into irreducible representations. It is perhaps not surpris- 
ing, that with the appropriate conditions on the imbedding of the subgroup, the class of (in our 
case derived functor) modules is preserved in the restriction from H to G. 

While the algebraic methods of derived functor modules, in particular the cohomologically 
induced representations, provide a very strong tool for the theoretical investigations of unitary 
representations of reductive Lie groups, it has been difficult to work with concrete models of 
these modules. It is however exactly these models that we use in this paper, as outlined above; 
the fact that one may consider these modules as Taylor expansions of appropriate differential 
forms, indicates that it should be natural to study these Taylor expansions along submani- 
folds - and when these submanifolds are natural for the subgroup for which one wants to do the 
branching law, there arises a useful link between the algebraic branching and the geometry of the 
imbedding of the subgroup. It is our hope, that this idea (that we carry out in some relatively 
small examples) will have a broader use in deciding the possible candidates for representations 
occuring in an admissible branching law. 

Here is the general setting that we consider: Let G be a semisimple linear connected Lie 
group with maximal compact subgroup K and Cartan involution 9. Suppose that a is another 
involution so that a -9 = 9 -a and let H be the fixpoint set of a in G. Suppose that L = is the 
centralizer of an elliptic element x G i(g n t)) and let q = [ © u, = q fl f) be the corresponding 
^-stable parabolic subgroups. Here we use as usual gothic letters for complex Lie algebras and 
subspaces thereof; a subscript will denote the real form, e.g. Qo- We say that pairs of parabolic 
subalgebras q, q^ which are constructed this way are well aligned. For a unitary character A 
of L we define following Vogan/Zuckerman the unitary representations Aq(X). 

In this paper we consider the example of the group G = 5'L(4, M). There are two G-conjugacy 

classes of skew symmetric matrices with representants Qi = \ "1 ^ t i Q'^ J ^ 



^ -J J \ J 

where ~ 0^ ) ' respectively H2 be the symplectic subgroups defined by these 

matrices and H[, H2 the centralizer of Qi, respectively Q2- All these subgroups are fix point 
sets of involutions cjj, i = 1,2 and cj-, i = 1,2 respectively. 

The matrix Q2 has finite order, it is contained in all subgroups Hi and iQ2 G defines 
a ^-stable parabolic subalgebra q of sl{A, C) and also ^-stable parabolic subalgebras q[,j = q fl f)i 
of t}i, respectively qija = 1 H {)2 of f)2- Its centralizer L in 5L(4,M) is isomorphic to GLi(2,C) = 
{T € GL(2,C)| I det(T)| = 1}. The parabolic subgroups q, qfj^ as well as q, q(,2 are well aligned. 

We consider in this paper the unitary representation Aq of G corresponding to trivial cha- 
racter A. Its infinitesimal character is the same as that of the trivial representation. The 
representation Aq was studied from an analytic point of view by S. Sahi [21j . Since the Aq has 
nontrivial (g, i^)-cohomology and is isomorphic to a representation in the residual spectrum, 
this representation is also interesting from the point of view of automorphic forms. See for 
example |23] . We show in this paper that the restriction of Aq to Hi and H[ is a direct sum 
of irreducible unitary representations, where as the restriction to H2 and H2 has continuous 
spectrum. We also determine explicitly the restriction of Aq to the subgroups Hi and H[ and 
conclude that for all unitary (f)i, Hi D i^)-modules V the dimension of Hom^jj^ //^n^)(Aq, V) is 
at most 1. 
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If we interpret 5'L(4, M) and 5*^(2, M) as Spin(3,3) and Spin(2,3) then these branching laws 
can in some sense be considered as supporting the conjectures by B. Gross and D. Prasad [3] for 
the restriction of Vogan packets of representations of SO{n, n) to representations of SO{n — l, n). 

The paper is organized as foUows: After introducing ah the notation in Section [2] we prove 
in Section [3] using a result of T. Kobayashi, that the restriction of to Hi and H[ is a direct 
sum of irreducible unitary representations, whereas the restriction to H2 and H'2 does have 
a continuous spectrum. This discrete/continuous alternative, see [12], is one of the deep results 
that we invoke for symmetric subgroups. 

We do not attempt in this paper to say anything about the continuous spectrum, and we 
mainly focus on the admissible situation, so the alternative is really admissible/non-admissible. 

In Sections H] and [5] we determine the representations of H\ respectively of H[ that appear 
in the restriction of to Hi respectively H'^ and show that it is a direct sum of unitary 
representations of the form ylqn(,^(/x) respectively Aq^f,/^ (//'), each appearing with multiplicity 
one. The main point is here, that we find a natural model in which to do the branching law, 
based on the existence results of T. Kobayashi; and also following experience from some of his 
examples, where indeed derived functor modules decompose as derived functor modules (for the 
smaller group). 

In Section [6] we introduce pseudo dual pairs. This allows us to find another interpretation 
of the restrictions of ^q to the pseudo dual pair Hi, H2. This notion turns out to be extremely 
useful for analyzing the spectrum in the admissible situation, and combined with our idea of 
restricting a cohomologically induced module gives the complete branching law. In Section [7] we 
recall some more examples of branching laws. 

In Section [8] we formulate a conjecture about the multiplicity of representations in the restric- 
tion of representations of semisimple Lie groups G to subgroups H, which are centralizers of 
involutions. If the restriction of Aq to -ff is a direct sum of irreducible representation of H we 
expect that there is a ^-stable parabolic subalgebra of H so that all representations which 
appear in the restriction are of the form A^ndj.) and that a Blattner-type formula holds. See 
the precise conjecture at the end of Section [HI where we introduce a natural generalization of 
previously known Blattner-type formulas for the maximal compact subgroup. 

In Section [9| we see how these results may be used to construct automorphic representations 
of Sp(2,M) and GL(2,C) which are in the discrete spectrum for some congruence subgroup. 
For Sp{2, M) these representations are in the residual spectrum, whereas for GL{2, C) these 
representations are in the cuspidal spectrum. We expect that our methods extend to other 
situations with similar applications to automorphic representations; and we hope the point of 
view introduced here also will help to understand in a more explicit way the branching laws for 
semisimple Lie groups with respect to reductive subgroups. 

2 Notation and generalities 

2.1. Let G be a connected linear semisimple Lie group. We fix a maximal compact subgroup K 
and Cartan involution 9. Let H he a ^-stable connected semisimple subgroup with maximal 
compact subgroup = KCiH. We pick a fundamental Cartan subgroup = ■ A^ of H. 
It is contained in a fundamental Cartan subgroup C = T • A of G so that = T n H and 
A^ = AnH. The complex Lie algebra of a Lie group (as before) is denoted by small letters and 
its real Lie algebra by a subscript o. We denote the Cartan decomposition by 0o = ffi P- 

Definition. Let q and be 0-stable parabolic subalgebras of g, respectively P|. We say that 
they are well aligned if q^ = q n f). 

We fix Xo in t^. Then i Xq defines well aligned 0-stable parabolic subalgebras q = [ u and 
q-f^ = [-f^ u-^ = q n f) of g respectively f); for details see page 274 in [TO] . 
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We write L and L for the centrahzer of xq in G and in H respectively. For a unitary 
character A of L we write for the restriction of A to . 

2.2. For later reference we recall the construction of the representations ^q(y), V an irreducible 
(q,L n K) module. We follow conventions of the book by Knapp and Vogan [10] (where much 
more detail on these derived functor modules is to be found - this is our standard reference) 
and will always consider representations of L and not of the metaplectic cover of L as some 
other authors. We consider U{q) as right U{(\) module and write V'^ = V ® A*°Pu. Let Pl be 
a L n ii'-invariant complement of [ H £ in I. We write vq = Now we introduce the derived 
functor modules as on page 167 in [1^, recalling that this formalizes Taylor expansions of certain 
differential forms. Since all the groups considered in the paper are connected we use the original 
definition of the Zuckerman functor [26] and do not use the the Hecke algebra -R(0, K) to define 
the representations A^iV). Consider the complex 

^ Homini^(t/(0), Hom(A V, V^))k ^ 
^ Homini^(C/(0), Hom(AVG, V^))k ^ Hom^^ni^ ([/(fl), Hom(AVG, V^))k ^ • • • • 

Here the subscript K denotes the subspace of i^-finite vectors. We denote by T{x,U{-)) an 
element in Homj;,nii'(^^(0)5 Homc(A'"~-^rG', V'^))k- The differential d is defined by 

n 

d T{x, U{Xi A ^2 A • • • A Xn)) = ^{-IfTiXiX, U{Xi A X2 A ■ ■ ■ Xi ■ ■ ■ A 

1=1 

n 

+ ^i-'^y^'T{x, XiU{Xi AX2A---Xi--- AXn)) 

i=l 

+ Y.i-'^y^'T{x, U{Pra [Xi, Xj] A Xi A X2 A • • • 1, • • • X, • • • A 

i<j 

where x € U{g), Xj e re and Pre is the projection onto re along [ fi 6. Let s = dim(u n £) and 
let X be the infinitesimal character of V. If 

fc4^0{O,-l,-2,-3,...} for aeA(u), 

where ( , ) denotes the Killing form of g, then the cohomology is zero except in degree s and 
if V is irreducible this defines an irreducible {(U (q) , K)-module A^{V) in degree s (8.28 in |10|). 
By (5.24 in [T^, see also the remark/example on page 344) the infinitesimal character of ^q(l^) 
is X + /'(u) (usual shift of the half-sum of all positive roots in u). 

If V is trivial the infinitesimal character of ^q(V^) is pc and we write simply Aq. Two 
representations A^ and A^^i are equivalent if q and q' are conjugate under the compact Weyl 
group Wk- 

For an irreducible finite dimensional {q^,L^ H /ir)-module we define similarly the 

([/((}), K^)-modules A^niV^"). 

2.3. Let H be the fix point set of an involutive automorphism cj of G which commutes with the 
Cartan involution 6. We write Qo = f)o ©So for the induced decomposition of the Lie algebra. 
T. Kobayashi proved [12] that the restriction of Aq to H decomposes as direct sum of irreducible 
representations of H if A^ is /C^-admissible, i.e. if every i^-'^-type has finite multiplicity. If A^ is 
discretely decomposable as an ([)o,i^nif)-module we call an irreducible (f)o, -ff nK)-module 

an H-type of Aq if 

Bom^i^^^KH)iTT",Aq)^0 

and the dimension of ilom.(^^^ j^H'^{ir^ , Aq) its multiplicity. 
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We have ( = © [Rs. Put = uHi). The representation of on u is reducible and as l^- 
module u = U'^©(un5). Let q = [©u be the opposite parabohc subgroup. Then t) = t^©u^©u'^ 
and (u n s) © u n s is a [^-module. As an [^-module g = f) © (u n s) © (I n s) © (u n s). 

2.4. Now let G = SL{4,M). The skew symmetric matrices 
Qi = [i ^j) and Q2=(o 5 

with J = ( ^ ^ ) represent the conjugacy classes of skew symmetric matrices under G. They 



.1 . 

define symplectic forms also denoted by Qi and Q2- 

Let Hi, respectively H2, be the 6'-stable symplectic subgroups defined by Qi, respectively Q2- 
These subgroups are fix points of the involutions 

'Tiig) = Q^ ■ {9~T ■ Q7\ i = h2. 

Since Qi and Q2 are conjugate in GL(4, R), but not in 5L(4,R), the symplectic groups Hi 
and H2 are not conjugate in 5^(4, R). 

Let H[ and H2 be the fix points of the involutions 

<r'i{g)=Q^■g■Q;'■ 

Both groups H[ and H2 are isomorphic to 

GLi(2,C) = {T e GL(2,C)| |det(T)| = 1}, 
but they are not conjugate in 5L(4, M). 

2.5. We fix xq = Q2- It has finite order and is contained in n?=i ^'^^ {^i=i^'i- Now 
ixo G is defines a stable parabolic subalgebra q of s/(4, C) and also ^-stable well aligned 
parabolic subalgebras q^ of the subalgebras f). Its centralizer L = Lx^ in S'L(4,]R), the Levi 
subgroup, is isomorphic to GL{2, C) = H'2. For a precise description of the parabolic see page 586 
in [10]. 



We have 

L = H'2, K^^ = KnH2 = KnL, = KnH'2 = KnL 

and 



Let Aq be the representation holomorphically induced from q which has a trivial infinite- 
simal character. This representations is a subrepresentation of a degenerate series representa- 
tion induced from a one dimensional representation of the parabolic subgroup with Levi factor 
S{GL[2,M) X GL(2,M)) and thus all its i^-types have multiplicity one. See [21] for details. 

The next proposition demonstrates how different imbeddings of the same subgroup (sym- 
plectic res. general linear complex) gives radically different branching laws. 

Proposition 2.1. 

1. The restriction of A,^ to Hi and to H[ is a direct sum of irreducible representation each 
appearing with finite multiplicity. 

2. The restriction of Aq to H2 and to H2 is not admissible and has continuous spectrum. 
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Proof. Since K^^ = K^'i and K^^ = it suffices by T. Kobayashi's Theorem 4.2 in [12] 
to show that is K^^ admissible. We will prove this in the next section. 

To prove (2) it suffices to show by Theorem 4.2 in [12] that j4q is not K^^ admissible. This 
is proved also in the next section. ■ 

Remark. We will prove later in the paper that the representations of Hi have at most multi- 
plicity one in the restriction of Aq. 

3 The restriction of Aq to K D Hi, i = 1,2 

We use in this section the notation introduced on pages 586-588 in [lOj . 

3.1. The Cartan algebra to of so(4, M) consists of 2 by 2 blocks ( ^ 1 down the diagonal. 

We have a ^-stable Cartan subalgebra i)o = to (B Oo where Qq consists of the 2 by 2 blocks 



Xj 
Xj 



also down the diagonal. We define G f)* by 



^ iyj Xj I ~ 



and /j G fl* by 

' V J ' 

Then the roots A(u) of (Pi,u) are 

ei + 62 + (/l - /2), ei + 62 - (/l - /2), 261, 262 

and a compatible set of positive roots A'^([) of ([), [) are 

61 - 62 + (/l - /2), 61 - 62 - (/l - /2)- 

The roots qi = 6i + 62, 02 = 6i — 62 are compatible positive roots of the Lie algebra 6 with 
respect to t. 

The highest weight of the minimal i^T-type of is A = 3(6i + 62). See page 588 in [10]. All 
other iC-types are of the form 

A + mi(6i + 62) + 2m26i, mi,m2GN. 

K^'^ = KnL is isomorphic to U{2). We assume that the positive root of (t, 6^^) is Q2 whereas 
the ai can be identified with a character of the center of K^^ . So the restriction of each K-type 
with highest weig ht (m + 3)ai to K^^ is a sum of characters dai with — (m + 3) < d < (m + 3). 
Hence A^ is not ii'^^-finite and thus by T. Kobayashi's theorem A^ is not a discrete sum of 
irreducible representations of H2. This proves the second claim of Proposition 12.11 

The groups Hi and H2 are conjugate under the outer automorphism which changes the sign 
of 62. Hence the simple positive root of (t, ) can be identified with ai and 02 can be identified 
with a character of the center of K^^ (see Fig. [1]). A J^-type with highest weight A + mi{ei + 
62 ) + 2m2 61 is a tensor product of a representation with highest weight (3+mi+m2,3+mi+ m2 ) 
and a representation with highest weight (777-2, —m.2). Its restriction to K^^ is a direct sum of 
representations with highest weights (3 + 77ii + 7772 + 7,3 + 7711 + 70,2 — 7), —7712 <i< 7772. Fig. [2] 
shows the highest weights of the -fT^^ -types for the restriction of Aq to K^^ . Their multiplicities 
are indicated by a number. 
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Figure 1. 



Figure 2. 



Thus is il'-'^i -finite. This completes the proof of the first assertion of Proposition 12. li 

Remark. A second series of representations is obtained if we define another ^-stable parabohc 
subalgebra q' using the matrix Qi G g instead of Q2. We obtain a representation A^^i which is 
not equivalent to A^. The same arguments as in the previous case prove that restriction of A^i 
to Hi does not have a purely discrete spectrum whereas the restriction to H2 is a direct sum of 
irreducible unitary representations. 

The representation of S'L~''/~(4, M) (determinant ±1) obtained by inducing representa- 
tion j4q is irreducible and its restriction to 5^(4, M) is equal to Aq © A^i. Hence the restriction 
of ^q to Hi does not have discrete spectrum. 



4 The restriction of Aq to the symplectic group Hi 

In this section we determine ffi-types of A^. Our techniques are based on homological algebra 
and the construction of an "enlarged complex" whose cohomology computes the restriction. We 
introduce it in 4.1 for semisimple connected Lie groups H and connected reductive subgroups H. 
Then we will compute the restriction to Hi by restricting A^ to a subgroup conjugate to Hi. 
The motivation for this "enlarged complex" or "branching complex" is the same as when one is 
restricting holomorphic functions to a complex submanifold, and identifying the functions with 
their normal derivatives along the submanifold. In our case we are working with (formalizations 
of) differential forms satisfying a similar differential equation, so it is natural to try to identify 
them with their "normal derivatives" ; this is what is formalized in our definition. As it turns out, 
with the appropriate conditions (well aligned parabolic subgroups, vanishing of the cohomology 
in many degrees, and the non-vanishing of explicit classes corresponding to small iiT-types) we 
can indeed make the calculation of the branching law effective, at least in the examples at hand. 

4.1. We define another complex for a semisimple connected Lie group G and a connected 
reductive subgroup H satisfying the assumptions of 2.3. Let Ca^ be the one dimensional 
representation of defined by A (8> A*°p(u n s). Then 
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Consider the complex 



(4.1) 



Here r/;- = n f) and da is defined analogously to the differential d in 2.2. As a left U{{^)- 
module 

C/(g) =Q0 [/((,), 

where Q is the symmetric algebra ^(s). (See [ini 2.56].) We have 

HomLnii-nH(C^(0),Hom(AVH,c5^^))ft:nH = ^oiULr^Kr^uiQ ®U{\]),mm{A\HX\^))KnH 
= HomLn/^nH(f/(f)), Hom(AV//, Q* (g) C^^))KnH. 

[/ (g) acts on the enlarged complex from the right and a quick check shows that dn also commutes 
with this action and therefore we have an action of U{q) on the cohomology of the complex. 
We also consider the "large complex" L* 

(Homj^nLnH(f/(0), Hom(AVG, C^))/rnH, d). 
We have = rjy © (u n s) © {pi H 5), and so 

aVg = ®i+k=i rn A\un 5® PL n 5). 
Using this we can define a "pull back" map of forms 

pb*^ : RomLnKnHiUil)),Bomi/^rH,Q* ®C{j)KnH 
RouiKnLnHiU (g), Hom(AVG, C{))KnH- 

The pullback map commutes with the right action of i7(f)) and induces a map of complexes. 
This is the main observation we use in order to analyze the action of H on the cohomologically 
induced module. 

4.2. For the rest of this section we assume that G = S'L(4,M). We will show that there exists 
a symplectic subgroup, which we denote by Hf conjugate to Hi by an element w, so that the 
pullback induces a nontrivial map in cohomology. Since the restriction of depends only on 
the conjugacy class of Hi this determines the restriction. 
Since 
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by abuse of notation we will also write Hi and H[ for the groups defined by the skew symmetric 
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for symmetric matrices X and Y. 
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Recall that q is defined by 



( 



i J 
i J 



) 



G 



and that = f)i ©Si. We need the fine structure of the parabolic relative to the symmetric 
subgroup, in order to compare the cohomology of these complexes during the branching. 

Lemma 4.1. Under the above assumptions 

a) [ n {}i is isomorphic to sl{2, M) © iM and dim(u fl = 3; 

b) the representation of L (1 Hi acts by a nontrivial character fii with differential (ei + 62) 
on the one dimensional space uflSi; 

c) [flsi is a direct sum of the trivial representation and the adjoint representation 0/ In f)i. 

d) un6 = untn[)i has dimension 1. 

Proof. We have 



The nilradical of a parabolic subalgebra with this Levi subalgebra has dimension 3. 

The dimension of [ H f)i H 6 is 2. Hence the dimension of u fl t fl {}i is 1. On the other hand the 
dimension of [ fl Ms 4. So the dimension of u fl Ms 1. Since un6n[)i Cuflfiwe have equality. 

uHsi is in the roots spaces for roots ei + 62 + (/i — /2) and ei + 62 — (/i — /2)- Hence [Pl f)i n6 
acts on u n Si by ei + 62. 

[ n f)i acts on the 4 dimensional space [ fl Si via the adjoint representation. ■ 

The representation of L H Hi on the symmetric algebra 5((u n Si) © (u n Si) © ([ fl Si)) is 
isomorphic to a direct sum of representations /i"^ (8) fJ^i"^^ <S) ad^^ with ni,mi,ri G N. These 
powers of /ii will label the constituents in the branching law; it will also be sometimes convenient 
to think of their differentials in additive notation. 

Now it is important to note, that the parameter Xhi (8) /^"^ , < ni is in the good range [lOj 
and thus the representation on the cohomology in degree 1 = dim(un6n of the complex L*jj_^ 
has composition factors isomorphic to 

Anf)i(A//i © /^i')' 

where < ni. In particular Aqnfji(A//J is an i(rni?i)-submodule module of the cohomology 
of L^^. 

Proposition 4.2. Aqn(jj(A//^) is a composition factor of the restriction of to [\)i,K r\ Hi). 

Proof. Note that dim(u H t) = dim(u fl £ fl and that 1 = dim(u fl 6) is the degree in which 
the complexes defining the representations respectively Aqnf,^(Aj^^) both have nontrivial co- 
homology [To] . Considering the complex defining Aq as a subcomplex of the "large complex" L* 
the pullback pb^^ of forms defines a (f)i, K n f/'i)-equi variant map 



/ a b X \ 

—6 a X 

y —a b 

V y -b -a I 



A^ffi~=o^qnf,i(/ir®AHi). 
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Recall the definition of the X-module 7^|^(A) from V.5.70 in [TO]. We have bottom layer 
maps of ^o-modules. 

and 

13{Xh^) : Agnt,^{XH^) T^KnHii^Hi), 

where Aq is the trivial character of L f] K. These maps are defined by the inclusion of of 
complexes and hence of forms. See Theorem V.5.80 and its proof in [10]. The minimal X- types 
of Aq, respectively i^^^-type of ^qn[j^(A/fJ are in the bottom layer. 

On the other hand we have an inclusion of complexes (the notation in analogy with the case 
in 4.1, now for the case where we take G = K) 

vWh^^k ■■ Hom/^nLnifi(f/(^),Hom(AXrGn«nf)i),c5^^^))/^nHi 
^ HomA^nLn/fi(C/(«),Hom(A*(rG n«),C^J)i^nHi. 

But C^^ = C^^ and rcntrifii = tq^^ and so using a forgetful functor we may consider 

Homi^nL(C/(«) , Hom( [re n I) , ))k 
subspace, respectively subcomplex, of 

RomKnLnH,iUit),Bom{A\rGritnl)i),C{^))Knm 

= RouiKHLnH, {Uit n f}i), Hom(A*(rG D^Dlji), Qh0 C{^^))Knm , 

where Qh is the symmetric algebra of the complement of f)i in t. Note that K (iHi and K is 
again a symmetric a pair and so we have a bottom-layer map for the representation 

Since the representation 7^^(Ao) is irreducible restricted to K n Hi this map is an isomor- 
phism. ■ 

Definition. We call Aqn^ii^Hi) the minimal i^i-type of Aq. 

Theorem 4.3. The representation A^ restricted to Hi is the direct sum of the representations 
each occuring with multiplicity one, namely 

Proof. By the proof of the lemma ^qpfjiC'^-ffi) is a submodule of the restriction of A^ to the 
symplectic group Hi. Its minimal K^^-type is also a minimal ^C-type of Aq and hence occurs 
with multiplicity one. Hence ^qnfji i^Hi ) is a -fTi-type of A^ with multiplicity one. 

The minimal K^^-type of ^qnfji (A) has highest weight A + 3ei + 3e2. The roots of u fl Pii Pip 
are 2ei , 2e2 . Applying successively the root vectors to the highest weight vector of the minimal 
K^^-type of j4qn(j^(A) we deduce that Aqn(,i(A) contains the K^^-types with highest weight 
((3 + 2ri)ei + (3 + 2r2)ei + A), ri,r2 G N. Theorem 8.29 in [10] show that all these ET^i -types 
have multiplicity one. Fig. [3]shows the i^^^-type multiplicities of ^q(A/fJ. 

Note that we are here using quite a bit of a priori information about the derived functor 
modules for the smaller group; on the other hand, the branching problem has essentially been 
reduced to one for compact groups, i^-type by iT-type. 
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Figure 3. 

The Borel subalgebra of t fl {)i acts on the one dimensional space u n Si by a character ni 
with differential (ei + 62)- Let y 7^ be in u H 5i and f 7^ a highest weight vector of the 
minimal ii'-types of Aq. Then Y"^ • u 7^ is also the highest weight of an i^-'^i-type of highest 
weight (3 + n)ei + (3 + n)e2 of Aq. 

Let X]. 7^ be in u n The linear map 

Ts : C/(0) ^ aVg ® 

which maps 1 to Xk ® C^^^ is non-zero in cohomology and its class [T^] is the highest weight 
vector of the minimal ii'-type. But 

so may consider 

Y-T, G Homini^n//i(C^(f)i),Hom(AV^,,,5l ®C^^^ 

Hence 7^ [F • T^] = y • [T,] G ^qn(,i(^i Ai^J and thus ^qnt)i(AHi /ii) is an i^i-type of ^q. 
The same argument shows that j4qn[j^(A/f^ (g) /i^), n G N, is a ffi-type of A^. 

Now every X-type with highest weight (n, n) has multiplicity n — 2 and is contained in 
exactly n — 2 composition factors. The multiplicity computations in Section [3] now show that 
every composition factor is equal to ylqn(,^(A/f^ ® for some n. See Fig. [H ■ 

Remark. Another proof of Theorem 14.31 can be obtained using Proposition 15.11 and the ideas 
of 6.1. 

By Proposition 8.11 in [10] for any (g, X)-module X we have 

Homg,j^(X,Aq) = Hom(,i^nL(i^.(u,X),C«), 

where Hsiu^X) is the Lie algebra homology as defined in [10] and s = dim(un6). Thus we have 
a "Blattner type formula" for the ifi-types of Aq. 
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Corollary 4.4. Xei y he an irreducible {t)i, K^^)-module. Then 

dimHomf,^_i^n//i(^, A) = X] *^™^°™(tnf)i,i^^inL)(-f^i(i^ f^i' ^)' n Si) ® Cj^J. 

i 

Remark. The Hi{-, •) on the right refers to homology in degree one. The maximal Abelian split 
subalgebra ai in [ n f)i are the diagonal matrices. So the parabolic subgroup of the Langlands 
parameter of the i?i-types of Aq is the so called "mirabolic" , i.e. the maximal parabolic subgroup 
with Abelian nilradical. The other parts of Langlands parameter can de determined using the 
algorithm in |10j . 

We consider in Theorem 14.31 the restriction of a "small" representation of Spin(3,3) to 
Spin(2,3) similar to the restriction of "small" discrete series representations of SO{n+ l,n) to 
SO{n,n) considered by B. Gross and N. Wallach in [3]. It would be interesting to see if their 
techniques could be adapted to the problem discussed in the paper. 

5 Restriction of Aq to the group H'^ 

In this section we describe //{-types of Aq using the same techniques as in the previous section. 
5.1. For H[ we consider the complex 

and the map 

pb*^, : Homin/<nH((f^(f)'i),Hom(AV^,,g* C^^) 

KnH[)KnH[ 
^0inKnLnH[ {U (fl), Hom(AVG, C{))KnH[ ■ 

We write g = [)[ ©s'^. The intersection u H s'^ is 2-dimensional and the representation of 
the group of LCi H[ on u n s'^^ is reducible and thus a sum of 2 one dimensional representations 
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Xi ffi X2- The weights of these characters are 2ei and 2e2- So the symmetric algebra 5'(u n s'^) 
is a direct sum of one dimensional representations of L n H[ with weights 2miei + 2/71262- 
In the cohomology in degree 1 of the complex L*^, we have composition factors 

with < ni,n2. In particular ^qpfi'^l-^/zj) is an (f}'^, X n //{)-submodule module of the cohomo- 
logy in degree 1. 

Proposition 5.1. ^qnfj'^ (^^//j) ^-^ composition factor of the restriction of to (f)'^,i7| fl K). 

Proof. The maximal compact subgroups of Hi and H[ are identical. Thus dimu H 6 fl f)i = 
dimu n 6 n f)i = 1 and the minimal if- type is irreducible under restriction to K^^. Thus the 
same argument as in Lemma |4. 1 1 completes the proof. ■ 

Definition. We call ^qnfj'^ (^^//j) the minimal H[ type of Aq. 

Theorem 5.2. The representation A^ restricted to H[ is the direct sum of the representations 
each occurring with multiplicity one, namely 

Their minimal KriH[ -types have highest weights {3-\-mi+m2+i,3+mi+m2—i), — m-2 <i< m2. 

Proof. The proof is the same in the previous section where we proved that the representations 
^qnf)'^ (-^//; (g) (g) X2^); ni,n2 G N appear in the restriction of the ^q to H[ 

The K n H[ -types of all unitary representations of GL(2,C) have multiplicity one. If the 
minimal K D //(-type has highest weight hei + l2e2 — 2, then the highest weights of the other 
K n //(-types are (/i -|- j)ei -|- {I2 + j)e2- Multiplicity considerations oi K f^ //(-types of A^ 
conclude the proof. ■ 

Remark. Using Proposition 14. 2t 4.1 and the ideas of 6.1 we can obtain another proof of this 
theorem. 

Fig. [5] shows the decomposition into irreducible representations. The highest weights of the 
K n //(-types of a composition factors lie on the lines. For each highest weight there is exactly 
one composition factor which has a K H //(-type with this weight as a minimal K n //(-type. 

We have again a "Blattner-type formula" for the //( types. 

Corollary 5.3. Let V be an irreducible {Ij'i, {K f] H[))-module, then 

dimHom(,/^^n//(Cl^,^q) = X] ^™^°™([nf)i).-^n/f;nL(^i(u n f^;, F), 5'(u n s'l) C{^,). 

i 

5.2. All the //(-types ^qn(,/^(A//j (gXi^ ®X2^) of are simply unitarily induced principal series 
representations of GL(2,C). 

6 Pseudo dual pairs 

6.1. Suppose now that G is a reductive connected Lie group with maximal compact subgroup K, 
Cartan involution ^ of g. For an involution 
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Figure 5. Figure 6. 



commuting with 6 we define 

T' = Toe and H = G^, H' = G^' . 

Definition. We call H and H' a pseudo dual pair. 
Since 6, r and r' commute 

Lemma 6.1. Suppose that H and H' are a pseudo dual pair in G. Then 

1) Kr\H = Kr\H'; 

2) we have p = f ® p""'; 

3) [/(g) ^ 5(p^ )S(p^)C/(6) and so we can write every element in U{q) as a linear combination 
of terms of the form u;2^i^k where ujk € U{t), uji € U{\]) and uJ2 € U{i]'). 

Suppose now that vr be an irreducible unitary (g, A')-module. Then vr is K n i7-admissible if 
and only if it is ii'n //'-admissible and by Theorem 4.2 of [12] its restriction to is a direct sum of 
irreducible ([), Kni/)-modules if and only if tt is a direct sum of irreducible (f)', -fCn//')-modules. 

Suppose that tt \s H r\ K admissible and that T4 C vr is a minimal iT-type of vr. If tTq is 
a ([), X n i/)-module, which occurs in the restriction of tt to [) then 

TTo n u{^')Vk + 0. 

6.2. These observations allow us to better understand the restriction of to the pseudo 
dual pair Fi=5p(2,M), H{= GL{2,C) in S'L(4,M). The minimal K-type Vk of has highest 
weight 3ai and is also irreducible under KCiHi = KnH[. The (f)'^, -fCn//()-submodule generated 
by Vk is the minimal H[ type and is isomorphic to a spherical principal series representation. 
We draw a diagram of its K n //(-types using the same conventions as in the previous sections. 
The K n //(-types are on the black line in Fig. [6l 

Each KOHi = Kr)H[-type of this representation is the minimal KCiHi-type of an irreducible 
(f)i, /^ n //i)-module as indicated in Fig. [71 
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Figure 7. Figure 8. 




02 



Figure 9. 

The K n Hi types of the minimal A' n ifi)-module generated by the minimal ii'-type 
have multiplicity one and are indicated by the dots in Fig. [HI 

Each of the K n f^i-types of this K n iJi)-module is the minimal K n Hi = K r\ i7(-type 
of a (f)'^, K n H[)-modn\e in the restriction of to -ffj, as illustrated in Fig. [9l 

7 More branching 

In this section we sketch the restriction to Hi and H[ of a representation ^q(Ai) with a parame- 
ter A which is no longer in the weakly fair range and hence the representation is no longer 
irreducible and has a composition series of length 2. We indicate a procedure to use our previous 
techniques to compute the restriction of both composition factors of ^q(Ai) to Hi and H[. Using 
the local isomorphisms 



SL(4,IR) ~ 50(3,3), 5p(2,M) ~ 50(2,3), 5L(4,M) n GL(2,C) ~ S0(2) x S0(l,3) 
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we obtain a different proof of a result by T. Kobayashi and B. 0rsted of the branching of the 
minimal representation vr of S0{3, 3) to S0{3, 1) x S0{2) and to SO{3, 2) [TTJ. 

7.1. We use the notation introduced in 3.1. Using the conventions on page 586 in [10] we 
denote the character of L by A = 771(61 + 62 + 63 + 64). With this parametrization the representa- 
tion ^q(A) is irreducible and unitary for m > — 3 (see page 588 in [10]) and corresponds to 
the parameter m = 0. The representation Aq(A) for m = 3 is nonzero. 

To simplify the notation we denote the representation ^q(A) for A = m{ei + 62 + 63 + 64) 
by Aq(m). 

We consider now the representation Ai^{—3). This representation is nonzero and has a trivial 
K-type. It is not irreducible, but has 2 composition factors. One composition factor is Aq(— 2). 
The other composition factor is a unitarily induced representation ttq from one dimensional 
representation of a maximal parabolic subgroup. It is a ladder representation and the highest 
weights of its K-types are multiples of 61. See [l7] for 50(3, 3) or [l] for GL(4,R). 

The restriction of Aq{—3) to Hi D K are mi{ei + 62) + 2m2ei, 7711,7712 € N. 

7.2. The same arguments as in Sections [4] and O allows us to compute the restriction of Aq(— 3) 
to Hi and H'^ . In this case the minimal Hi type is a spherical representation t^q ^ and comparing 
the multiplicities of K n -ffi-types of vr^^ and of ttq we deduce that the restriction of ttq to H^ is 
irreducible and equal to tt^^ . A similar argument show that the restriction to H[ is direct sum 
of principal series representations of GL{2, C). 

Using the local isomorphism S'L(4,M) and S'0(3, 3) we obtain a new proof of the branching 
of ttq determined by B. 0rsted and T. Kobayashi in [T7] . 

8 A conjecture 

8.1. The examples in the previous section and the calculations in jl4j support the following 
conjecture: Let H be the connected fixpoint set of an involution a. We write again g = f) © s. 
Let Aq(A) be a representation, which is n ff- admissible and thus decomposes discretely, when 
restricted to H. Suppose q, q'* are defined by Xo G . Since j4q = j4p if q and p are conjugate 
under the compact Weyl group Wk we use the following 

Definition. Let yo € and let p, be well aligned parabolic subalgebras defined by i/o- 
We call the well aligned parabolic subalgebras p, p^ related to q, q^, if Xo and yo are conjugate 
by an element in the compact Weyl group Wk of K with respect to T. 

If Xo and yo are not conjugate by an element in the Weyl group Wkhh of {K^,T") then the 
parabolic subalgebras q^, p^ of H are not conjugate in H and thus we have up to conjugacy at 
most Wk/Whdk different pairs of well aligned pairs of 9 -stable invariant parabolic subalgebras 
which are related to q, q^. If G = 5^(4, R), H = Hi and (q, q n [}i) is the pair of well aligned 
parabolic subalgebras defined by xq = Q2, there there are at most 2 related pairs of well aligned 
parabolic subalgebras. 

We expect the following Blattner-type formula to hold for the restriction to H: 

Conjecture. There exists a pair p, p^ of well aligned 0-stable parabolic subgroups related 
to q, q^ so that every H-type V of A^ is of the form A^h{^i) for a character ^ of and that 

dimHomf,^^H(y, ^p) = ^ ^(-1)''-^ dimHomLnH(^i(u D [),V), S'{un 5) (g) CxJ. 

i j 

Remark. Some of the characters /.i in this formula may be out of the fair range as defined 
in [To] and hence reducible. 
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If H is the maximal compact subgroup K, then |Wx/W^Hnx| = 1) all related pairs of well 
aligned parabolic subalgebra are conjugate to q, and hence we get the usual Blattner formula 
(5.108b on page 376 in [TO]^. 

In the example discussed in this paper, G = 5L(4, M), the representation and H the 
symplectic group considered in Section HI all related pairs of well aligned parabolic subalgebras 
are conjugate to q, qjj^ and thus we obtain the Blattner type formula in Corollary 14.41 

9 An application to automorphic representations 

We use here our results to give different constructions of some known automorphic represen- 
tations of Sp{2,M) and GL{2,C). We first explain the ideas in 7.1 in a more general setting. 
Again we may consider restrictions, this time in the obvious way of restricting functions on 
locally symmetric spaces to locally symmetric subspaces. 

9.1. Assume first that G is a semisimple matrix group and T an arithmetic subgroup, H a semi- 
simple subgroup of G. Then Th = T Ci H is an arithmetic subgroup of H. Let C Lp'{G/T) 
be an irreducible (g, i(r)-submodule of Lp'{G/T). If / € 14- then / is a C°°-function and so we 
define fn as the restriction of f to H/Th- 

Lemma 9.1. The map 

RES^^: K ^ C^{H/Th) 
f ^ fH 

is an {[), K n H)-inap. 

Proof. Let ht = expitXn), ho € H. Then 

p{XH)f{ho) = j^f{hj'ho)t=o = j^fHiK'ho)t=o = p{XH)fHiho). ■ 

Suppose that the irreducible unitary (g, i^)-module vr is a submodule of Lp'{G/T) and that 
its restriction to i7 is a direct sum of unitary irreducible representations. 

Proposition 9.2. Under the above assumptions RES/f(7r) is nonzero and its image is contained 
in the automorphic functions on H/Th- 

Proof. Let fn be a function in RES//(7r). Then by Section [2] it is K D ff-finite and we may 
assume that it is an eigenfunction of the center of f/(f)). 

Let ll^lp = tT{g*g). Since sup\f{g)\ H^H"'' < oo, the same is true for fn and so fn is an 

automorphic function on H/Th- 

The functions in the (g, -fC)-module vr C Lp'{G/T) are eigenfunctions of the center of the 
enveloping algebra U{q) and are ET-finite, hence analytic. Thus if / is a i^-finite function in 
TT C L2(G/r) then there exists W G U{q) so that Wf{e) / 0. Hence RESi^(VF/) / 0. ■ 

Instead of restricting the automorphic function f to the orbit of e/T under H we may also 
consider the restriction to an orbit of 7/r, for rational 7. Since the rational elements are 
dense at least one of the restrictions is not zero. So following Oda we consider the restriction 
correspondence for functions on G/T to functions on Yig^/^ ^ 9^9~^- For rational g the 
intersection T n gTg~^ contains an arithmetic group T' and F' n is an arithmetic subgroup. 
For more details see for example page 55 in [2]. 

9.2. Now we assume that G = GL(4,M) and that F C GL(4, Z) is a congruence subgroup. The 
groups Fl = Fni/i and F'^ = Fni?( are arithmetic subgroups of 5^(4, M), respectively GL{2, C). 
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Recall the definition of the {q, K)-m.odule Aq from 3.1. It is a submodule of Lp'{Z\G /T) 
for r small enough where Z the connected component of the center of GL(4,R). We will 
for the remainder of this sections consider it as an automorphic representation in the residual 
spectrum [23]. Then RES//^(^q) and RESj|//(^q) are nonzero. Its discrete summands are 
contained in the space of automorphic forms. 

Theorem 9.3. The discrete summands of the two representations RES//j(.4q) respectively 
RESj:/'(^q) are subrepresentations of the discrete spectrum of L?'{Hi/T Hx), respectively 

Proof. All the functions in ^q decay rapidly at the cusps. Since the cusps of Hi/Thi are 
contained in the cusps of G/T this is true for the functions in RESi^j(^q). Thus they are also 
contained in the discrete spectrum. ■ 

For Sp{2,M) the representations constructed in the previous theorem were first described by 
H. Kim (see [9] and [22])- For GL(2,C) we obtain the stronger result 

Theorem 9.4. The representations in the discrete spectrum of JlES}j>^{Aq) are in the cuspidal 
spectrum of {H[ /Tjj/^ ) . 

Proof. By 5.2 the representations in the discrete spectrum of RESj:/j(„4q) are unitarily in- 
duced principal series representations and so by a result of Wallach they are in fact cuspidal 
representations. ■ 



The embedding of H[ = GL(2,C) into 5L(4,M) is defined as follows: Write g = A + iB with 
real matrices A, B. Then 



Since all the representations in the discrete spectrum of the restriction of Aq do have nontrivial 
(f), i^^)-cohomology with respect to some irreducible finite dimensional nontrivial representa- 
tion F we obtained the well known result [5l [20] . 

Corollary 9.5. There exists a congruence subgroup T C GL{2, Z[i]) and a finite dimensional 
non-trivial representation F of GL{2, C) so that 

W{r,F)^0 for i = l,2. 
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